In MPEG-1 and MPEG-2 the input video consists of image frames sampled at regular time intervals. This represents the finest temporal resolution of the input FIG. 1 shows a video sequence with fixed frame rates where the image frames are sampled at regular intervals. In the coded representation of the video sequence using the MPEG-1 and MPEG-2 standards the display order of the decoded frames is indicated by the temporal reference. This parameter appears at the picture header of the bitstream syntax. The value of this parameter is incremented by one for each decoded frame when examined in the display order.
In the H.263 standard it is possible to skip frames and thus decode a variable frame rate video sequence. However the sampling of frames is still fixed. Thus the temporal reference method used in MPEG-1 and MPEG-2 is still appropriate and only need to be modified such that the increment is not in steps of 1 but in steps of 1 plus the number of non-transmitted pictures at the input frame rate.
Currently work is being done in the area of coding video as separate objects in multiple video object planes. This represents a new dimension in the decoding and synchronization of the respective video objects. It is expected that these different video object planes may come from several sources and can have very different frame rates. Some of the objects may be rendered and have almost continuous temporal sampling rate. These video objects planes are combined into a composite image for display. Therefore some kind of synchronization is required for the composition. It is possible that the display frame rate is different from any of the frame rates of the video object planes. FIG. 2 shows an example of two video object planes having different and varying frame rates compared to one another. Even if a common frame rate can be found between the two video object plane, it is not automatic that this frame rate will be the same as the output frame rate of the compositor.
Hereafter we will refer to the problem in the video domain. However the same principles of this invention can be extended to the audio domain as well as combinations of the two.
From the above it is clear that the current state of the art does not cater for the synchronization of video object planes. The state of the art also does not provide a common temporal reference when the different video object planes have different frame rates which are not multiples of one another.
The first problem is how to provide a common local time base mechanism for each video object plane. This time base will have to be able to provide very fine temporal granularity and at the same time be able to cope with possibility of very long intervals between two successive instances of the video object planes.
The second problem is how to provide a mechanism for synchronizing video object planes having different frame rates.